System and method for multipurpose traffic detection and characterization

ABSTRACT

A method for tracking and characterizing a plurality of vehicles simultaneously in a traffic control environment, comprising: providing a 3D optical emitter; providing a 3D optical receiver with a wide and deep field of view; driving the 3D optical emitter into emitting short light pulses; receiving a reflection/backscatter of the emitted light, thereby acquiring an individual digital full-waveform LIDAR trace for each detection channel of the 3D optical receiver; using the individual digital full-waveform LIDAR trace and the emitted light waveform, detecting a presence of a plurality of vehicles, a position of at least part of each vehicle and a time at which the position is detected; assigning a unique identifier to each vehicle; repeating the steps of driving, receiving, acquiring and detecting, at a predetermined frequency; tracking and recording an updated position of each vehicle and an updated time at which the updated position is detected.

Notice: More than one reissue application has been filed for the reissueof U.S. Pat. No. 9,235,988. The reissue applications are applicationSer. No. 15/867,995 (the present application) filed Jan. 18, 2018 andSer. No. 17/412,092 filed Aug. 25, 2021.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a reissue application based on U.S. application Ser.No. 14/115,244 filed on Nov. 1, 2013, which previously issued as U.S.Pat. No. 9,235,998, which is a national stage entry of PCT/IB2013/051667filed Mar. 1, 2013, which claimed the benefit of provisional application61/605,896 filed on Mar. 2, 2012.

TECHNICAL FIELD

The present invention relates to a system and method for trafficdetection and more particularly to an optical system that detects thepresence, location, lane position, direction and speed of vehicles in atraffic zone using an active three-dimensional sensor based on thetime-of-flight ranging principle and an image sensor.

BACKGROUND OF THE ART

Growth in transportation demand has a major impact on traffic congestionand safety. To enhance the on-road safety and efficiency, majorinvestments in transport infrastructures, including capital, operationand maintenance, are made all over the world. Intelligent systemscollecting and disseminating real time traffic information is a keyelement for the optimization of traffic management.

Traffic monitoring can consist in different activities such as detectingthe presence of a vehicle in a specific zone, counting the number ofvehicles (volume), determining the lane position, classifying eachvehicle, determining the direction of travel, estimating the occupancyand determining the speed.

Other traffic surveillance applications such as electronic tollcollection and traffic enforcement require the same kind of informationwith a very high level of reliability.

In the United States, the FHWA has defined a vehicle classificationbased on 13 categories of vehicles from motorcycles, passenger cars,buses, two-axle-six-tire-single unit trucks, and up to a seven or moreaxle multi-trailer trucks classes. Several alternative classificationschemes are possible. Often, the aggregation of the FHWA 13 classes issplit into 3 or 4 classes. Other countries have their own way to definea classification for vehicles.

In the case of speed infringement, determining the position and thelane, measuring accurately the speed of a specific vehicle in amulti-lane high-density highway, and associating this informationwithout any ambiguity with the vehicle identified using an AutomaticLicense Plate Recognition (ALPR) system is quite challenging.

A red light enforcement system has comparable requirements. There is aneed for an automatic red light enforcement system but the highreliability required for this application is also challenging. Itimplies the detection of vehicles at specific locations, the tracking ofeach of these vehicles in dense traffic at the intersection, theidentification of each of these vehicles with the ALPR system, theconfirmation of a red light violation by a specific vehicle and thecollection of all information to support the issuance of a trafficviolation ticket to the registered owner of the vehicle without anyambiguity.

Different kinds of detectors are used to collect data for theseapplications. Intrusive detectors such as inductive loop detectors arestill common for detecting the presence of vehicles but have somedisadvantages such as lengthy disruption to the traffic flow duringinstallation and maintenance, inflexibility and inability to track avehicle. Cameras with video processing have some drawbacks notably forspeed measurement.

Radar technology is known to perform well for speed measurement but hassome limitations in terms of lateral resolution making difficult theassociation between a speed measurement and the identification of aspecific vehicle in dense traffic, for example, at an intersection.Radar technology presents difficulties in the correlation of a specificspeed measurement to a specific vehicle when two or more vehiclestraveling at different speeds simultaneously enter into the measurementbeam. This limitation has an impact for speed enforcement applications.In some countries, legislation requires that ambiguous situations simplybe discarded to reduce errors in the process. Installation of radartechnology for speed enforcement is demanding because it requiresadjusting the angle of the axis of the main lobe of emission in both thehorizontal and vertical directions with respect to the axis of the road,with accuracy typically less than one-half degree angle to limit thecosine effect.

Thus, there is a need for a method and system for reliable multipurposetraffic detection for traffic management and enforcement applications.

SUMMARY

According to one broad aspect of the present invention, there isprovided a method for tracking and characterizing a plurality ofvehicles simultaneously in a traffic control environment. The methodcomprises providing a 3D optical emitter at an installation heightoriented to allow illumination of a 3D detection zone in theenvironment; providing a 3D optical receiver oriented to have a wide anddeep field of view within the 3D detection zone, the 3D optical receiverhaving a plurality of detection channels in the field of view; drivingthe 3D optical emitter into emitting short light pulses toward thedetection zone, the light pulses having an emitted light waveform;receiving a reflection/backscatter of the emitted light on the vehiclesin the 3D detection zone at the 3D optical receiver, thereby acquiringan individual digital full-waveform LIDAR trace for each detectionchannel of the 3D optical receiver; using the individual digitalfull-waveform LIDAR trace and the emitted light waveform, detecting apresence of a plurality of vehicles in the 3D detection zone, a positionof at least part of each the vehicle in the 3D detection zone and a timeat which the position is detected; assigning a unique identifier to eachvehicle of the plurality of vehicles detected; repeating the steps ofdriving, receiving, acquiring and detecting, at a predeterminedfrequency; at each instance of the repeating step, tracking andrecording an updated position of each vehicle of the plurality ofvehicles detected and an updated time at which the updated position isdetected, with the unique identifier.

In one embodiment, the traffic control environment is at least one of atraffic management environment and a traffic enforcement environment.

In one embodiment, detecting the presence includes extractingobservations in the individual digital full-waveform LIDAR trace; usingthe location for the observations to remove observations coming from asurrounding environment; extracting lines using an estimate line and acovariance matrix using polar coordinates; removing observations locatedon lines parallel to the x axis.

In one embodiment, detecting the presence includes extractingobservations in the individual digital full-waveform LIDAR trace andintensity data for the observations; finding at least one blob in theobservations; computing an observation weight depending on the intensityof the observations in the blob; computing a blob gravity center basedon the weight and a position of the observations in the blob.

In one embodiment, the method further comprises setting at least onetrigger line location and recording trigger line trespassing data withthe unique identifier.

In one embodiment, the method further comprises setting the trigger linelocation relative to a visible landmark in the environment.

In one embodiment, detecting the time at which the position is detectedincludes assigning a timestamp for the detecting the presence andwherein the timestamp is adapted to be synchronized with an externalcontroller.

In one embodiment, the method further comprises obtaining aclassification for each detected vehicles using a plurality ofdetections in the 3D detection zone caused by the same vehicle.

In one embodiment, detecting the presence further comprises detecting apresence of a pedestrian in the environment.

In one embodiment, the part of the vehicle is one of a front, a side anda rear of the vehicle.

In one embodiment, emitting short light pulses includes emitting shortlight pulses of a duration of less than 50 ns.

In one embodiment, the 3D optical emitter is at least one of an infraredLED source, a visible-light LED source and a laser.

In one embodiment, providing the 3D optical receiver to have a wide anddeep field of view includes providing the 3D optical receiver to have ahorizontal field of view angle of at least 20° and a vertical field ofview angle of at least 4°.

In one embodiment, the method further comprises determining andrecording a speed for each the vehicle using the position and theupdated position of one of the instances of the repeating step and anelapsed time between the time of the position and the updated time ofthe updated position, with the unique identifier.

In one embodiment, the method further comprises using a Kalman filter todetermine an accuracy for the speed to validate the speed; comparing theaccuracy to a predetermined accuracy threshold; if the accuracy is lowerthan the predetermined accuracy threshold, rejecting the speed.

In one embodiment, the method further comprises retrieving a speed limitand identifying a speed limit infraction by comparing the speed recordedfor each the vehicle to the speed limit.

In one embodiment, the method further comprises providing a 2D opticalreceiver, wherein the 2D optical receiver being an image sensor adaptedto provide images of the 2D detection zone; driving the 2D opticalreceiver to capture a 2D image; using image registration to correlatecorresponding locations between the 2D image and the detection channels;extracting vehicle identification data from the 2D image at a locationcorresponding to the location for the detected vehicle; assigning thevehicle identification data to the unique identifier.

In one embodiment, the vehicle identification data is at least one of apicture of the vehicle and a license plate alphanumerical code presenton the vehicle.

In one embodiment, the vehicle identification data includes the 2D imageshowing a traffic violation.

In one embodiment, the method further comprises extracting at least oneof a size of characters on the license plate and a size of the licenseplate and comparing one of the size among different instances of therepeating to determine an approximate speed value.

In one embodiment, the method further comprises providing a 2Dillumination source oriented to allow illumination of a 2D detectionzone in the 3D detection zone and driving the 2D illumination source toemit pulses to illuminate the 2D detection zone and synchronizing thedriving the 2D optical receiver to capture images with the driving the2D illumination source to emit pulses to allow capture of the imagesduring the illumination.

In one embodiment, driving the 2D illumination source includes drivingthe 2D illumination source to emit pulses of a duration between 10 μsand 10 ms.

In one embodiment, the 2D illumination source is at least one of avisible light LED source, an infrared LED light source and laser.

In one embodiment, the 3D optical emitter and the 2D illumination sourceare provided by a common infrared LED light source.

In one embodiment, the vehicle identification data is at least two areasof high retroreflectivity apparent on the images, the detecting apresence includes extracting observations in the individual digitalsignals and intensity data for the observations, the method furthercomprising correlating locations for the areas of high retroreflectivityand high intensity data locations in the observations, wherein each thearea of high retroreflectivity is created from one of a retroreflectivelicense plate, a retro-reflector affixed on a vehicle and aretro-reflective lighting module provided on a vehicle.

In one embodiment, the method further comprises combining multiples onesof the captured images into a combined image with the vehicle and thevehicle identification data apparent.

According to another broad aspect of the present invention, there isprovided a system for tracking and characterizing a plurality ofvehicles simultaneously in a traffic control environment, the systemcomprising: a 3D optical emitter provided at an installation height andoriented to allow illumination of a 3D detection zone in theenvironment; a 3D optical receiver provided and oriented to have a wideand deep field of view within the 3D detection zone, the 3D opticalreceiver having a plurality of detection channels in the field of view;a controller for driving the 3D optical emitter into emitting shortlight pulses toward the detection zone, the light pulses having anemitted light waveform; the 3D optical receiver receiving areflection/backscatter of the emitted light on the vehicles in the 3Ddetection zone, thereby acquiring an individual digital full-waveformLIDAR trace for each channel of the 3D optical receiver; a processor fordetecting a presence of a plurality of vehicles in the 3D detection zoneusing the individual digital full-waveform LIDAR trace and the emittedlight waveform, detecting a position of at least part of each thevehicle in the 3D detection zone, recording a time at which the positionis detected, assigning a unique identifier to each vehicle of theplurality of vehicles detected and tracking and recording an updatedposition of each vehicle of the plurality of vehicles detected and anupdated time at which the updated position is detected, with the uniqueidentifier.

In one embodiment, the processor is further for determining andrecording a speed for each the vehicle using the position and theupdated position of one of the instances of the repeating step and anelapsed time between the time of the position and the updated time ofthe updated position, with the unique identifier.

In one embodiment, the system further comprises a 2D optical receiver,wherein the 2D optical receiver is an image sensor adapted to provideimages of the 2D detection zone; and a driver for driving the 2D opticalreceiver to capture a 2D image; the processor being further adapted forusing image registration to correlate corresponding locations betweenthe 2D image and the detection channels and extracting vehicleidentification data from the 2D image at a location corresponding to thelocation for the detected vehicle; and assigning the vehicleidentification data to the unique identifier.

In one embodiment, the system further comprises a 2D illumination sourceprovided and oriented to allow illumination of a 2D detection zone inthe 3D detection zone; a source driver for driving the 2D illuminationsource to emit pulses; a synchronization module for synchronizing thesource driver and the driver to allow capture of the images while the 2Ddetection zone is illuminated.

According to another broad aspect of the present invention, there isprovided a method for tracking and characterizing a plurality ofvehicles simultaneously in a traffic control environment, comprising:providing a 3D optical emitter; providing a 3D optical receiver with awide and deep field of view; driving the 3D optical emitter intoemitting short light pulses; receiving a reflection/backscatter of theemitted light, thereby acquiring an individual digital full-waveformLIDAR trace for each detection channel of the 3D optical receiver; usingthe individual digital full-waveform LIDAR trace and the emitted lightwaveform, detecting a presence of a plurality of vehicles, a position ofat least part of each vehicle and a time at which the position isdetected; assigning a unique identifier to each vehicle; repeating thesteps of driving, receiving, acquiring and detecting, at a predeterminedfrequency; tracking and recording an updated position of each vehicleand an updated time at which the updated position is detected.

Throughout this specification, the term “object” is intended to includea moving object and a stationary object. For example, it can be avehicle, an environmental particle, a person, a pedestrian, a passenger,an animal, a gas, a liquid, a particle such as dust, a pavement, a wall,a post, a sidewalk, a ground surface, a tree, etc.

Throughout this specification, the term “vehicle” is intended to includeany movable means of transportation for cargo, humans and animals, notnecessarily restricted to ground transportation, including wheeled andunwheeled vehicles, such as, for example, a truck, a bus, a boat, asubway car, a train wagon, an aerial tramway car, a ski lift, a plane, acar, a motorcycle, a tricycle, a bicycle, a Segway™, a carriage, awheelbarrow, a stroller, etc.

Throughout this specification, the term “environmental particle” isintended to include any particle detectable in the air or on the groundand which can be caused by an environmental, chemical or naturalphenomenon or by human intervention. It includes fog, water, rain,liquid, dust, dirt, vapor, snow, smoke, gas, smog, pollution, black ice,hail, etc.

Throughout this specification, the term “red light” is intended to meana traffic light (traffic signal, traffic lamp or signal light) which iscurrently signaling users of a road, at a road intersection, that theydo not have the right of way into the intersection and that they shouldstop before entering the intersection. Another color and/or symbol couldbe used to signal the same information to the user depending on thejurisdiction.

Throughout this specification, the term “green light” is intended tomean a traffic light (traffic signal, traffic lamp or signal light)which is currently signaling users of a road, at a road intersection,that they have the right of way into the intersection and that theyshould enter the intersection if it is safe to do so. Another colorand/or symbol could be used to signal the same information to the userdepending on the jurisdiction.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a betterunderstanding of the main aspects of the system and method and areincorporated in and constitute a part of this specification, illustratedifferent example embodiments. The accompanying drawings are notintended to be drawn to scale. In the drawings:

FIG. 1 is a functional bloc diagram of an example of the multipurposetraffic detection system showing its main components and the way theyare interconnected;

FIG. 2 is an example installation of the traffic detection system on theside of a 3-lane highway;

FIG. 3 shows an example installation of the traffic detection system ona gantry;

FIG. 4 shows the impact on the depth of a detection zone of the heightof installation of the system;

FIG. 5 shows an example casing for the multipurpose traffic detector;

FIG. 6 shows a top view of the detection zone on a 3-lane highway;

FIG. 7 shows a top view of the detection zone in a red light enforcementapplication;

FIGS. 8A and 8B are photographs showing example snapshots taken by theimage sensor with the overlay of the 3D sensor displaying a vehicle inthe detected zone with distance measurements;

FIG. 9A is a photograph showing an example snapshot taken by the imagesensor with the overlay of the 3D sensor at an intersection for redlight enforcement application and FIG. 9B is a graph of data acquired bythe detection system showing the range of detection of vehicles on 3lanes in Cartesian coordinates;

FIG. 10 is a top view of an example road side installation with thetracking system being installed next to a one-directional three-lanehighway and for which the detection zone is apparent and covers, atleast partly, each of the lanes, all vehicles traveling in the samedirection;

FIG. 11 is a top view of the example installation of FIG. 10 on whichfour vehicle detections are visible in some of the 16 separate channelswith simultaneous acquisition capability;

FIG. 12 is a top view of the example installation of FIG. 10 on which adetection is visible between two trigger lines;

FIG. 13 includes FIGS. 13A, 13B, 13C, 13D, 13E and 13F, in which FIGS.13A, 13C and 13E are photographs which show a few frames of vehicletracking when vehicles arrive at an intersection with a red light andFIGS. 13B, 13D, and 13F show a graph of data acquired by the detectionsystem for each corresponding frame;

FIG. 14 includes FIGS. 14A, 14B, 14C, 14D, 14E and 14F, in which FIGS.14A, 14C and 14E are photographs which show a few frames of vehicletracking when vehicles depart the intersection of FIG. 13 at the greenlight and FIGS. 14B, 14D, and 14F show a graph of data acquired by thedetection system for each corresponding frame;

FIG. 15 is a flowchart illustrating an example method for trackingseveral vehicles based on a space-based tracking disjoint;

FIG. 16 is a flowchart illustrating an example method for trackingseveral vehicles for a red-light enforcement application, this algorithmuses a space-based tracking joint;

FIG. 17 is a flowchart illustrating the selection of appropriatemeasures among the detections;

FIG. 18 shows an example segment extraction line for a long vehicle;

FIG. 19 is a state diagram illustrating the tracking system used withouta traffic light state;

FIG. 20 is a state diagram illustrating the tracking system used with atraffic light state;

FIG. 21 is a flowchart showing example steps performed to compute thevehicle position;

FIG. 22 is a flowchart showing example steps performed for objecttracking without a traffic light state;

FIG. 23 is a flowchart showing example steps performed for objecttracking with a traffic light state;

FIG. 24 is a flowchart illustrating an example classification process;

FIG. 25 includes FIGS. 25A, 25B and 25C which illustrate therelationship between the detections of a vehicle and its geometricfeatures of width and length;

FIG. 26 illustrates the direct geometric relationship between height ofthe vehicle and distance of vehicle detection;

FIG. 27 includes FIGS. 27A, 27B, 27C and 27D which show top view framesof a vehicle detected by the LEDDAR sensor;

FIG. 28 includes FIGS. 28A, 28B, 28C and 28D which show correspondingside view frames of the vehicle of FIG. 27;

FIG. 29 is a flowchart illustrating an example segmentation algorithmbased on a 3D bounding box;

FIG. 30 is a top view of an example scenario used for the analysis ofPosterior Cramer-Rao lower bound;

FIG. 31 is a graph showing theoretical performance of the trackingalgorithm given by the PCRB;

FIG. 32 includes FIGS. 32A, 32B, 32C and 32D in which FIG. 32A is aphotograph showing an example snapshot taken by the image sensor duringthe day, FIGS. 32B, 32C and 32D are photographs showing a zoom in onlicense plates in the snapshot of FIG. 32A;

FIG. 33 includes FIGS. 33A, 33B and 33C in which FIG. 33A is aphotograph showing an example snapshot taken by the image sensor atnight without any light, FIG. 33B is a photograph showing the same sceneas FIG. 33A taken by the image sensor at night with an infrared lightillumination, FIG. 33C is a photograph showing a zoom in on a licenseplate extracted from the image of FIG. 33B;

FIG. 34 includes FIGS. 34A, 34B, 34C and 34D in which FIG. 34A is aphotograph showing another example snapshot taken by the image sensor atnight with infrared light, FIG. 34B is a photograph showing a zoom in ona license plate extracted from the image of FIG. 34A, FIG. 34C is aphotograph showing an example snapshot taken by the image sensor with ashorter integration time at night with infrared light, FIG. 34D is aphotograph showing a zoom in on a license plate extracted from the imageof FIG. 34C; and

FIG. 35 is a photograph showing an example panoramic snapshot taken bythe image sensor using infrared illumination in which two vehicles arepresent in the detection zone and on which the overlay of the 3D sensoris shown with dashed lines.

DETAILED DESCRIPTION

Description of the Multipurpose Traffic Detection System

Reference will now be made in detail to example embodiments. The systemand method may however, be embodied in many different forms and shouldnot be construed as limited to the example embodiments set forth in thefollowing description.

The functionalities of the various components integrated in an examplemultipurpose traffic detection system 10 can be better understood byreferring to the functional block diagram shown in FIG. 1. The 3DOptical Emitter 12 (3DOE) emits short pulses of light, for example of alength less than 50 ns, within a predetermined zone. In the exampleembodiment, the 3DOE 12 is an IR LED illumination source determining aField-of-Illumination FOI_(3D) covering the 3D detection zone FOV_(3D).The optical source of the 3DOE can also be based on Laser technology.The horizontal angles of the FOI_(3D) and FOV_(3D) are wide enough tocover at least one lane. For example, a system with a horizontal FOI/FOVof 35° would be able to cover 3 lanes, each lane having a width of 3.5m, when installed at 15 m from the side of the detection zone.

An example mounting configuration of the multipurpose traffic detectionsystem 10 can be seen in FIG. 2, which depicts a schematic view of aroadway with 3 lanes being shown. The traffic detection system 10 isshown mounted on a pole 27 with an orientation towards trafficdirection. Pole 27 can be a new dedicated road infrastructure for thesensor installation or an already existing road infrastructurestreetlight assembly or other types of infrastructures like gantries orbuildings. This exemplary roadway comprises three adjacent traffic lanesfor vehicles. The traffic detection system is intended to detect anytype of objects that may be present within the predetermined 3Ddetection zone.

The mounting height of the traffic detection system 10 is, for example,between 1 to 10 m with a lateral distance from the nearest traffic laneof, for example, between 1 to 5 m. In FIG. 2, three vehicles travellingin the same direction on the traffic lanes enter in the 3D detectionzone. When the vehicles reach the 3D detection zone, the multipurposetraffic detection system is used for detection, localization,classification and measurement of the speed of the vehicles through thezone. The system can also be installed over the roadway on a gantry asshown in FIG. 3. The system can also detect vehicles traveling inopposite directions.

The detection system can be installed at different heights, from theground up to 10 m. FIG. 4 shows the impact of the installation height onthe longitudinal length of the detection zone. With a fixed startingdistance of detection, the longitudinal length of the detection zonewill be shorter with a system installed higher. The vertical angles ofthe FOI_(3D) and FOV_(3D) have to be wide enough to detect and trackvehicles over several meters, for example over at least 8 m. Forexample, a system installed at a height of 3.5 m with a vertical FOI/FOVof 6° and a detection zone beginning at 15 m from the detector will havea detection zone depth of approximately 13 m.

Referring back to FIG. 1, part of the light diffusively reflected by thevehicles and objects in the FOI_(3D) is directed towards the collectingaperture of the 3D Optical Receiver 14 (3DOR) for its 3D opticaldetection and subsequent conversion into digital waveforms. To bedetected, an object should appear within the FOV_(3D) of the 3DOR, whichis defined by its optics as well as by the dimensions of its opticallysensitive device. The 3DOR is composed of one or more optical lenses,multichannel optical detectors, for example photodiode arrays, an analogfrontend and analog-to-digital converter. Usually, the channels aredigitalized in parallel and the system implements a full-waveform signalprocessing of the signal waveforms generated by the plurality of opticaldetection channels.

The multipurpose traffic detection system provides a good accuracy interms of lateral resolution and is less dependent on the angle ofinstallation than Radar technology.

In FIG. 1, the 2D Optical Receiver 16 (2DOR) is at least one imagesensor, for example a CMOS or CCD (including front end and ADconversion) which provides images of the portion of the roadway areathat encompasses or overlaps at least a section of the FOI_(3D) of the3DOE and the FOV_(3D) of the 3DOR. The 2DOR will be used duringinstallation, to transmit video data, and, for some applications, tohelp identify vehicles using, for example, Automatic License PlateRecognition (ALPR) techniques. For applications requiring vehicleidentification, the requirement for the image sensor in terms ofresolution is high. An external image sensor or camera can also be usedfor this function. The average size of a character on a license plate isbetween 50 mm to 80 mm. It takes at least 16 pixels per character(height) to obtain good results with an Optical Character Recognition(OCR) processing within an ALPR system. Based on that criterion, theidentification of a license plate of a vehicle circulating on a 3-lanehighway (3.5 m×3 m) requires an image sensor with a least 5 Mpixels(2.5K×2K). High resolution image sensors are expensive. One way toreduce the cost is to use at least two image sensors each with lowerresolution and to combine the information coming from both images usingimage stitching techniques. The synchronization, acquisition and imageprocessing are performed by Control and processing unit 22.

The 2D Illumination 18 (2DI) is an optical source emitting infraredand/or visible light. The 2DI can be embedded in the sensor enclosure orcan be an external module. In one example embodiment, the optical sourceof 2DI 18 is at least one LED. LEDs are efficient and the FOI can beoptimized with optical collimators and diffusers. The pulse width of2DOE can be in the range of 10 μs to 10 ms and can be synchronized withthe image capture (integration time) of the image sensor(s). Forvehicles traveling at high speed, the integration time can be in therange of 500 μs and less. A vehicle moving at 150 km/h will travel 21 cmin 500 μs.

A single set of infrared LEDs can be used for both the 3DOE and 2DOE.Very high-short intensity pulses (for example <50 ns) for 3D detectioncan be mixed with longer pulses (for example 10 μs to 10 ms) for 2Dsensor(s). The LEDs can have a wavelength between 800 and 1000 μm, forexample.

Source Driver Electronics (SDE) 20 uses dedicated electronics fordriving the 3DOE 12 with current pulses having peak amplitude andduration suitable for effective implementation of the optical rangingprinciple on which the operation of the multipurpose traffic detectionsystem is based. A pulsed voltage trig signal forwarded by the Controland Processing Unit 22 commands the generation of each current pulse bythe drive electronics. The operating conditions and performancerequirements for the multipurpose traffic detection system call for theemission of short optical pulses having a duration in the range of 5 to50 ns, for example. Depending on the repetition rate at which the pulsesare emitted, the duty cycle (relative ON time) of the optical emissioncan be as low as 0.1%. In order to get the desired peak optical outputpower for the radiated light pulses, any lowering of the peak drivelevel of the LEDs or Laser can be compensated by mounting additional LEDor Laser sources in the 3DOE 12 and appropriately duplicating theirdrive electronics.

The SDE 20 can also drive 2D illumination with current pulses havingpeak amplitude and duration suitable for effective illumination of thescene for the 2DOR 16. A pulsed voltage trig signal forwarded by theControl and Processing Unit 22 commands the generation of each currentpulse by the drive electronics. The operating conditions and performancerequirements for the multipurpose traffic detection system call for theemission of 2D optical pulses having a duration in the range of 10 μs to10 ms, for example.

The SDE 20 can control and receive information from 3DOE and 2Dillumination about the intensity of the current pulse, LEDs/Lasertemperature, etc.

All of these modules exchange data and receive commands and signals fromthe control and processing unit 22. The Control and processing unit 22can include digital logic (for example by a Field-Programmable GatedArray (FPGA)) for pre-processing the 3D raw data and for thesynchronization and control, a memory, and a processing unit. Theprocessing unit can be a digital signal processing (DSP) unit, amicrocontroller or an embarked personal computer (PC) board as will bereadily understood.

The primary objective of the 3D full-waveform processing is to detect,within a prescribed minimum detection probability, the presence ofvehicles in a lane that is mapped to a number of adjacent detectionchannels. Because of the usual optical reflection characteristics of thevehicle bodies and of various constraints that limit the performances ofthe modules implemented in a traffic detection system, the opticalreturn signals captured by the 3DOR are optimized by acquisitionshifting techniques, accumulation techniques and filtering andcorrelation technique to enhance the signal-to-noise ratio (SNR) of theuseful signal echoes and detect a digital replica of the pulse emittedby the 3DPE. The properties (peak amplitude, shape, time/distancelocation) of the useful features present in the waveforms should remainideally unchanged during the time period required to capture a completeset of waveforms that will be averaged. This condition may cause issueswhen attempting to detect vehicles that move rapidly, this situationleading to signal echoes that drift more or less appreciably fromwaveform to waveform. The detrimental impacts of this situation can bealleviated by designing the traffic detection system so that it radiateslight pulses at a high repetition rate (e.g., in the tens to hundreds ofkHz range). Such high repetition rates will enable the capture of a verylarge number of waveforms during a time interval sufficiently short tokeep the optical echoes associated to a moving vehicle stationary.Detection information on each channel can then be upgraded, for examplebetween a few tens to a few hundred times per second. For example, witha multipurpose traffic detection system using a frame rate at 200 Hz, acar at 250 km/h would have moved forward by 35 cm between each frame.

The Control and processing unit 22 has numerous functions in theoperation of the multipurpose traffic detection system, one of thesebeing the calibration of the system. This calibration process can bedone by connecting a remote computer to the Control and processing unit22 and communicating using a Power management and data Interface 24.

During normal operation of the multipurpose traffic detection system,Power management and data Interface 24 receives information from theexternal controller (including parameters like a speed limit) and alsoallows the Control and processing unit 22 to send data. The data sentcan be related to the detection of each vehicle and can compriseinformation such as an accurate timestamp of the detection timesynchronized with the external controller, a unique identifier (IDnumber), the lane and position of the vehicle (lateral and longitudinal)for each trigger event, the position of the vehicle in an image, videostreaming, identification by ALPR, speed, classification, weatherinformation, etc., to the external controller.

In another embodiment, part of the process and algorithms can beintegrated in the external controller which receives the raw data fromthe Control and processing unit by the Power Management and Interface.

Several types of interfaces can be used to communicate with the externalcontroller: Ethernet, RS-485, wireless link, etc. Power over Ethernet(PoE) may be used for its simplicity of connection including power, dataand distance (up to 100 m).

The data information can also be stored in memory and retrieved later.

Power management and data Interface 24 can also send electrical triggersignals to synchronize events like the detection of the front or therear of a vehicle at a specific position to other devices like anexternal camera, an external illuminator or other interface and externalcontroller.

The Power Supply Management and Data Interface 24 can also be useful intransmitting images and videos to an external system or network to allowa remote operator to monitor different traffic events (ex.: accident,congestion, etc.). Video compression (ex.: MPEG) can be done by aprocessor to limit the bandwidth required for the video transmission.

The four optical modules can be rigidly secured to the attachmentsurface of an actuator assembly (not shown). The modules can then pivotin a controlled manner about up to three orthogonal axes to allow aprecise alignment of their common line of sight after the multipurposetraffic detection unit has been installed in place and aligned in acoarse manner. The fine-tuning of the orientation of the line of sightis, for example, performed remotely by an operator via a computer deviceconnected to the multipurpose traffic detection system, for examplethrough PoE or a wireless data link.

FIG. 1 also shows a functional bloc labeled Sensors 26 for measuringdifferent parameters. The internal temperature in the system enclosurecan be monitored with a temperature sensor which can be used to controla heating/cooling device, not shown. The current orientation of thesystem can be monitored using an inclinometer/compass assembly. Suchinformation may be useful for timely detection of the line of sight thatmay become misaligned. The sensor suite may also include anaccelerometer for monitoring in real-time the vibration level to whichthe system is submitted to as well as a global positioning system (GPS)unit for real-time tracking of the location of the system and/or forhaving access to a real-time clock.

FIG. 5 shows an example casing with a window 28 for the multipurposetraffic detection system. The casing can house a more or less completesuite of monitoring instruments, each of them forwarding its output datasignals to the control and processing unit for further processing orrelay. In other configurations of the casing, lateral sections can beintegrated to protect the window from the road dust.

Use, Set-Up, Basic Principles, Features and Applications

FIG. 6 shows a top view of an installation of the multipurpose detectionsystem. The multichannel 3DOR detects vehicles present within atwo-dimensional detection zone, the active nature of the trafficdetection system provides an optical ranging capability that enablesmeasurement of the instantaneous distances of the detected vehicles fromthe system. This optical ranging capability is implemented via theemission of light in the form of very brief pulses along with therecordal of the time it takes to the pulses to travel from the system tothe vehicle and then to return to the system. Those skilled in the artwill readily recognize that the optical ranging is performed via theso-called time-of-flight (TOF) principle, of widespread use in opticalrangefinder devices. However, most optical rangefinders rely on analogpeak detection of the light pulse signal reflected from a remote objectfollowed by its comparison with a predetermined amplitude thresholdlevel. In the present system, the traffic detection system numericallyprocesses the signal waveform acquired for a certain period of timeafter the emission of a light pulse. The traffic detection system cantherefore be categorized as a full-waveform LIDAR (Light Detection andRanging) instrument. The system analyses the detection and distancemeasurements on several 3D channels and is able to track severalvehicles at the same time in the detection zone. The system candetermine the lane position, the distance from the detector and thespeed, for each individual vehicle.

As can be seen in FIG. 6, the detection system 10 is installed at areference line 60, has a wide FOV 61, has a large and wide detection andtracking zone 62 covering several lanes and several meters of depth anddetects several vehicles on several lanes in a roadway.

The detection system can be configured with two trigger positions. Thefirst trigger 63 is set in the first section of the detection zone andthe second trigger 64 is set a few meters away, in this case close tothe end of the detection zone. In this example, a first vehicle 65 wasdetected when entering the detection zone on lane 1, was tracked, wasdetected at the position of the first trigger 63, was continuouslytracked and is now being detected at the position of the second trigger64. Information about its lane position, speed, etc., can be constantlysent or can be sent only when the vehicle reaches pre-establishedtrigger positions. A second vehicle 66 was detected when entering thedetection zone on lane 2, was tracked, was detected at the position ofthe first trigger 63, and is continuously tracked until it reaches theposition of the second trigger 64. A third vehicle 67 was detected whenentering the detection zone on lane 3, was tracked, is detected at theposition of the first trigger 63, will continue to be tracked and willreach the position of the second trigger 64.

The detection system has the capability to identify, track and sendinformation about multiple vehicles at the same time and its multiplereceiver channels greatly reduce the cosine effect for speedmeasurement.

The system can capture several snapshots using the 2DOR at differentlevels of illumination using the 2DOE. Information about each vehicle(date/hour of an event, speed, position, photographs and identificationbased on Automatic License Plate Recognition) can be sent to theexternal controller. This is useful for applications like trafficmanagement (for vehicle detection, volume, occupancy, speed measurementand classification), speed enforcement, red light enforcement, etc. Thesystem can be permanently or temporarily installed. It can even be amobile system, for example a system installed on a vehicle.

An example of configuration for Red Light Enforcement is shown in FIG.7. The capability of the system to detect, track, determine the laneposition, measure the speed and take photographs (or videos) for eachvehicle several meters away from the stop bar has great value for thisapplication. Red light enforcement applications require the detection ofa vehicle entering an intersection when the traffic light is at the redstate and the automatic capture of several images of the vehicle as itcrosses the stop bar and runs the red light. The detection system needsto provide evidence that a violation occurred without ambiguity.

For most applications, detection rates should be high, for example ofthe order of 95% and more (without occlusion), and false detectionsshould occur only very rarely. Images and information about the date andtime of the infraction will allow the authorities to transmit a trafficinfraction ticket. Identification of the driver and/or owner of thevehicle is generally made by the authorities using the information fromthe license plate of the vehicle. Since speed information is available,speed infractions can also be detected when the traffic light is green.As will be readily understood, the detection system can also be used forother detection applications such as stop line crossing and railwaycrossing.

In FIG. 7, the detection system is installed on the side of the road atan example distance of 15 to 25 m from the stop bar 70. The detectionand tracking zone 71 starts few meters before the stop bar 70 and coversseveral meters after the bar, allowing a large and deep zone fordetecting and tracking any vehicle on several lanes (three lanes in thatexample), at different speeds (from 0 to more than 100 km/h), at a rateof up to ten vehicles detected per second. The detection system can takeseveral images of a red light infraction including, for example, whenthe vehicle is located at a predetermined trigger distance, for exampleat first trigger 72 when the back of the vehicle is close to the stopbar 70 and at second trigger 73 when the back of the vehicle is fewmeters away from the stop bar 70. Optional detection of the laneposition is useful when a right turn on red is allowed at theintersection.

Speed enforcement is another application that requires providingevidence that a speed violation occurred. The correlation between thedetected speed and the actual vehicle guilty of the infraction needs tobe trustworthy. Sufficient information should be provided to allowidentification of the vehicle owner, using information from the licenseplate, for example. The capability of the detection system to measurethe speed of several vehicles at the same time with high accuracy and tomake the association between each speed measurement and the specificidentified vehicle is useful for traffic enforcement applications. Thisis made possible by, among others, the multiple FOV, the robustness andaccuracy of the sensor and the capability to store several images of aviolation.

The detector can store speed limit data (which can be different for eachlane) and determine the occurrence of the infraction.

The detector can be mounted on a permanent installation or can also betemporary, provided on a movable tripod for example. Detectors can alsobe installed at the entry and at the exit of a point-to-pointenforcement system allowing the measurement of the average speed of avehicle by determining the amount of time it takes to displace thevehicle between the two points. The position of each vehicle and itsclassification are also information that the detector can transmit tothe external controller. In some countries, lane restriction can bedetermined for specific vehicles, such as trucks for example.

Moreover, the multipurpose traffic detection system can fulfill morethan one application at a time. For example, the system used for trafficmanagement near an intersection can also be used for red lightenforcement at that intersection.

Methods for Alignment and Detection of the Traffic Detection System

A method that allows a rapid and simple alignment step for themultipurpose traffic detection system after it has been set in place isprovided.

FIGS. 8A and B show examples images of a roadway captured by the 2DORduring the day. The image is overlaid with the perimeters of a set of 16contiguous detection zones of the 3DOR. In FIG. 8A, a vehicle present inthe first lane 32 would be detected by several adjacent channels at arespective detected distance between 17.4 m to 17.6 m (see the numbersat the bottom of the overlay). In FIG. 8B, the vehicle is detected inthe second lane 34 between 24.0 m to 24.4 m. Note that the overalldetection zone is wide enough to cover more than two lanes. In somesituations depending on the context of the installation, some objects oreven the ground can be detected by the system but can be filtered outand not be considered as an object of interest.

FIG. 9A shows a photograph of a red light enforcement applicationinstallation. Some channels detect echo back signals from the ground(see the numbers at the bottom of the overlay) but the system is able todiscriminate them as static objects. FIG. 9B is a graph showing a topview of the 3D 16 field of view of a road with 3 lanes. In a Cartesiancoordinate system, if the detection system represents the origin, thehorizontal direction from left to right is taken as the positive x-axisand represents the width of the 3 lanes in meters, and the verticaldirection from bottom to top is taken as the positive y-axis andrepresents the longitudinal distance from the sensor. To facilitationinstallation, the installation software will indicate the beginning andthe end of the detection zone by showing a detection line as seen inFIG. 9B.

Multi-Vehicle Simultaneous Detection and Tracking for PositionDetermination, Speed Measurement and Classification

FIG. 10 shows a top view of an example road facility equipped with amultipurpose traffic detection system 10. The system 10 mounted on anexisting traffic infrastructure is used to illuminate a detection zone42. In this example, the mounting height is between 1 and 10 m with adistance from the road between 1 and 5 m. In FIG. 10, the vehicles 46travel in lanes 43, 44 and 45 in a direction indicated by arrow Athrough the detection system illumination zone 42. The detection system10 is used for detecting information of the rear surface of vehicles 46coming in the illumination zone 42. The detection system 10 is based onIR LED illumination source with a multiple field-of-view detector.

In FIG. 11, the 16 fields of view 52 covering a section of the road areshown. In a Cartesian coordinate system, if the detection systemrepresents the origin 49, the horizontal direction from left to right istaken as the positive x-axis 50, and the vertical direction from bottomto top is taken as the positive y-axis 51 then, each 3D detection 53gives the distance between an object and the sensor.

FIG. 12 shows the system in an example configuration with two triggerlines 56 and 57 located at a distance from the sensor between 10 and 50m, for example. The two trigger lines 56 and 57 are configured by theuser. Blob 55 illustrates a detectable vehicle rear. When the blobreaches the trigger line, the system returns a trigger message.

FIG. 13 and FIG. 14 show example data for vehicle tracking in thecontext of traffic light enforcement. Thanks to a projection of thefield-of-view of the detection system on the real 2D image, therelationship between the top view (FIGS. 13B, 13D, 13F) and the scene(FIGS. 13A, 13C, 13E) is made apparent. The 3D detections arerepresented by dots in the top views. In this example, a small diamondin the top views shows the estimated position of the rear of eachvehicle based on the 3D detections. In this example, the small diamondrepresents the middle of the rear of the vehicle. The distance ofdetection is indicated under each detection channel in the scene image.The amplitude of the detection is also indicated below the distance ofdetection. On the top view, thin lines define the limits of the trackingarea and dotted lines define two trigger lines configured by the user.When entering this area, a new vehicle is labeled with a uniqueidentifier. In each frame, its estimated position is shown using a smalldiamond. As shown, the interactions between vehicle detections aremanaged by the tracking algorithm allowing distinguishing vehicleslocated in the detection area.

FIG. 15 shows the steps performed during the execution of an exampletracking algorithm. At step 80, the tracking algorithm selects thereliable measurements located on the road. At step 81A, the genericKalman Filter for tracking a variable number of objects is used. At step82, a road user classification based on geometric features is computed.Finally, step 83 sends to each frame, a message with position, speed,class and trigger if necessary for the vehicles located in the detectionzone.

FIG. 16 shows the steps performed during the execution of the trackingalgorithm if the traffic light state 85 is known. Steps 80/800, 82 and83 are unchanged. However, step 81B is different because the additionalinformation allows working in a space-based tracking joint.

The selection of relevant measures 80 is described in FIG. 17. At step100 the tracking algorithm reads the available observations. At step101, the tracking algorithm removes each detection that is not locatedon the road. Step 101 is followed by step 102 where the trackingalgorithm recognizes lines by a feature-based approach. Step 103eliminates the points located on lines parallel to the x-axis 50 withthe aim of extracting the characteristics relating to the side(s) ofvehicles and to keep only the objects having a “vehicle rear signature”.

The estimation of a line based on the covariance matrix using polarcoordinate 102 is illustrated in FIG. 18. This estimation is based onfeature extraction. The strength of the feature-based approach lies inits abstraction from data type, origin and amount. In this application,line segments will be considered as a basic primitive which later servesto identify and then remove the side of vehicles. Feature extraction isdivided into two sub-problems: (i) segmentation to determine which datapoints contribute to the line model, and (ii) fitting to give an answeras to how these points contribute.

The polar form is chosen to represent a line model:x cos α+y sin α=r

where −π<α≤π is the angle between the x axis and the normal of the line,r≥0 is the perpendicular distance of the line to the origin; (x, y) isthe Cartesian coordinates of a point on the line. The covariance matrixof line parameters is:

${{cov}\left( {r,\alpha} \right)} = \begin{bmatrix}\sigma_{r}^{2} & \sigma_{r\;\alpha} \\\sigma_{r\;\alpha} & \sigma_{\alpha}^{2}\end{bmatrix}$

FIG. 19 shows a state diagram for the 3D real-time detectionmulti-object tracker. The core of the tracker 91A is based on a KalmanFilter in all weather and lighting conditions. The observation model 90is illustrated in FIG. 21 which presents an example method to computethe vehicle position by weighting each 3D observation according to itsheight amplitude. This method permits to improve the accuracy of theestimated position with respect to using only the x and y Cartesianpositions.

Expression 301 computes the blob position as follows:P_(blob)=Σ_(n=1) ^(N)π^(n)·P^(n)

where π^(n) is the intensity weight for the observation n, nϵ{1, . . . ,N}, and N is the number of observation grouped together. Step 301 isfollowed by computing the observation weight depending on the intensityat step 302.

The function 300 normalizes the weight π^(n) according to the amplitudeA^(n) of the observation P^(n):

$\pi^{n} = \frac{A^{n}}{{\Sigma A}^{n}}$

The state evolution model 92 is represented by the classical modelcalled speed constant. Kinematics model can be represented in a matrixform by:p_(k+1)=F·p_(k)+G·V_(k), V_(k)˜N(0,Q_(k))

where p_(k)=(x_(obs),{dot over (x)}_(obs),y_(obs),{dot over (y)}_(obs))is the target state vector, F the transition matrix which models theevolution of p_(k), Q_(k) the covariance matrix of V_(k), and G thenoise matrix which is modeled by acceleration.

$F = \begin{bmatrix}1 & {\Delta T} & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & {\Delta T} \\0 & 0 & 0 & 1\end{bmatrix}$ $G = \begin{bmatrix}\frac{{\Delta T}^{2}}{2} & 0 \\{\Delta T} & 0 \\0 & \frac{{\Delta T}^{2}}{2} \\0 & {\Delta T}\end{bmatrix}$ $Q_{k} = \begin{bmatrix}\sigma_{x}^{2} & 0 \\0 & \sigma_{y}^{2}\end{bmatrix}$

The equation observation can be written as:Z_(k)=H·p_(k)+W_(k), W_(k)˜N(0,R_(k))

where Z_(k)=(x_(obs) _(k) ,y_(obs) _(k) )^(t) is the measurement vector,H the measurement sensitivity matrix, and R_(k) the covariance matrix ofW_(k).

$H = \begin{bmatrix}1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 0\end{bmatrix}$ $R_{k} = \begin{bmatrix}\sigma_{{obs}_{x}}^{2} & 0 \\0 & \sigma_{{obs}_{y}}^{2}\end{bmatrix}$

The state space model 93A is based on probabilistic framework where theevolution model is supposed to be linear and the observation model issupposed to be Gaussian noise. In a 3D image, the system state encodesthe information observed in the scene, e.g. the number of vehicles andtheir characteristics is x_(k) ^(N)=(p_(k) ^(N), l_(k) ^(N)) with N asthe number of detected vehicles, where p_(k) ^(N) denotes the 2Dposition of object N at iteration k, l_(k) ^(N) gives identification,age, lane and the object classification.

FIG. 20 shows a state diagram for 3D real-time detection multi-objectjoint tracker. The core of 91B is based on a Kalman Filter whichaddresses the issue of interacting targets, which cause occlusionissues. When an occlusion is present, 3D data alone can be unreliable,and is not sufficient to detect, at each frame, the object of interest.If the algorithm uses the traffic light state 85, occlusions can bemodeled with a joint state space model 93B. The multi-object jointtracker includes a multi-object interaction distance which isimplemented by including an additional interaction factor in the vehicleposition. The state space model 93B encodes the observations detected inthe scene, e.g. the number of vehicles, the traffic light state and theinteraction between the vehicles located in the same lane byconcatenating their configurations into a single super-state vector suchas: X_(k)=(O_(k), x_(k) ¹, . . . , x_(k) ^(N)) with O_(k) the size ofstate space at iteration k and x_(k) ^(N)=(p_(k) ^(N), l_(k) ^(N)) thestate vector associated with the object N, where p_(k) ^(N) denotes the2D position of the object N at iteration k, l_(k) ^(N) givesidentification, age, lane, class, traffic light state and the objectinteraction.

Before integrating measures into the filter, a selection is made by atwo-step procedure shown in FIGS. 22 and 23: first at step 400validation gate, then at step 401A/B data association. The validationgate is the ellipsoid of size N_(z) (dimension of vector) defined suchas:θ^(t)·S⁻¹·θ≤γ

where θ^(t)=Z_(k)−

is the innovation, S the covariance matrix of the predicted value of themeasurement vector and γ is obtained from the chi-square tables forN_(z) degree of freedom. This threshold represents the probability thatthe (true) measurement will fall in the gate. Step 400 is followed bystep 401A/B which makes the matching between a blob and a hypothesis.Then, (i) consider all entries as new blobs; (ii) find the correspondingentries to each blob by considering gating intervals around thepredicted position of each hypothesis, (iii) choose the nearest entry ofeach interval as the corresponding final observation of each blob. Atstep 402, the tracking algorithm uses a track management module in orderto change the number of hypothesis. This definition is: (i) if,considering the existing assumption, there occurs an observation thatcannot be explained, the track management module proposes a newobservation; (ii) if an assumption does not find any observation after500 ms, the track management module proposes to suppress the assumption.In this case, of course, an evolution model helps to guide state spaceexploration of the Kalman filter algorithm with a prediction of thestate. Finally, step 403 uses a Kalman framework to estimate the finalposition of the vehicle.

In a 3D image, the system state encodes the information observed in thescene, the number of vehicles and their characteristics is X_(k)=(O_(k),x_(k) ¹, . . . , x_(k) ^(N)) with O_(k) the size of state space (numberof detected vehicles) at iteration k and x_(k) ^(N)=(p_(k) ^(N), l_(k)^(N)) the state vector associated with object N, where p_(k) ^(N)denotes the 2D position of object N at iteration k, l_(k) ^(N) givesidentification, age, lane and the object classification. Step 90 and 92are unchanged.

FIG. 24 shows the steps performed during the execution of theclassification algorithm. At step 500, the algorithm checks if a line isdetected in the 3D image. If a line is detected, step 500 is followed bystep 501 which computes vehicle length. Vehicle length is defined as theoverall length of the vehicle (including attached trailers) from thefront to the rear. In order to calculate the length, two differentpositions are used: X₀ and X₁ . . . X₀ is given by the position of thefirst detected line and X₁ is given by the trigger line 1 (for example).Once the speed has been estimated, the vehicle length l can bedetermined such as:

l[m]=s[m/S]*(X₁(t)[s]−X₀(t)[s])−(X₁(x)[m])−X₀(x)[m])+Seg[m]+TH[m] wheres is the vehicle speed, Seg is the length of the detected line and TH isa calibration threshold determined from a large dataset.

If the line is not detected at step 500, step 500 is followed by step502 which computes the vehicle height. The vehicle height is estimatedduring the entry into the sensor field of view. As shown in FIG. 26, fora known configuration of the detection system, there is a directgeometric relationship between the height of a vehicle 601 and thedetection distance 600. The accuracy 602 is dependent on the half-sizeof the vertical FOV angle 603. Height measurement is validated if theaccuracy is lower than a threshold.

Finally, step 502 is followed by step 503 which computes the vehiclewidth. Over the vehicle blob, let (y_(l), x) be leftmost pixel and(y_(r), x) be the rightmost pixel in the vehicle blob for a given x.Then the width w of the object is determined from the following formula:w=|y_(r)−y_(l)|

FIGS. 25A, 25B and 25C show a result of vehicle classification based onthe classification algorithm. For example, in FIG. 25A, theclassification result is a heavy vehicle; in FIG. 25B, it is afour-wheeled lightweight vehicle and in FIG. 25C, it is a two-wheeledlightweight vehicle. The information from the detection system isflexible and can be adapted to different schemes of classification. FIG.25 illustrates graphically the basic elements of the concept of anobject-box approach which is detailed below and in FIG. 27 and FIG. 28.

The object-box approach is mainly intended for vehicles because thisapproach uses the vehicle geometry in a LEDDAR image. The vehicles arerepresented by a 3D rectangular box of detected length, width andheight. The 3D size of the rectangular box will vary depending on thedetections in the FOV. FIGS. 27A, 27B, 27C and 27D show top view framesof a vehicle detected by the LEDDAR sensor. FIGS. 28A, 28B, 28C and 28Dshow corresponding side view frames of the vehicle of FIG. 27.

FIGS. 27A, 27B, 27C, 27D and FIGS. 28A, 28B, 28C, 28D show the changing3D size of the rectangle 701 for four example positions of a vehicle 702in the 3D sensor FOV 703. When a vehicle 702 enters the 3D sensor FOV703, two detections are made on the side of the vehicle (see FIG. 27A)and one detection is made for the top of the vehicle (see FIG. 28A). The3D rectangle is initialized with a length equal to 4 m, a width of 1.5 mand a height O_(Hm) given by:O_(Hm)=H_(s)−dist*tan(θ)

where H_(s) is the sensor height 704, dist is the distance of thedetected vehicle and θ is sensor pitch.

FIG. 27B and FIG. 28B represent detections when the vehicle isthree-fourths of the way in the detection FOV. Eight side detections areapparent on FIG. 27B and one top detection is apparent on FIG. 28B. Thedimensions of the 3D rectangle are calculated as follows:

The width is not yet adjusted because the vehicle back is not yetdetected.O_(l)(k)=max(L₂−L₁,O_(l)(k−1))O_(h)(k)=max(O_(Hm),O_(h)(k−1))

where the points of a segment are clockwise angle sorted so L₂ is thepoint with the smallest angle and L₁ is the segment-point with thelargest angle. O_(l)(k) and O_(h)(k) are respectively the current lengthand height value at time k.

FIG. 27C and FIG. 28C represent detections when the back of the vehiclebegins to enter in the detection FOV. Eight side detections and two reardetections are apparent on FIG. 27C while one detection is apparent onFIG. 28C. The dimensions of the 3D rectangle are calculated as follows:O_(l)(k)=max(L₂−L₁,O_(l)(k−1))O_(h)(k)=max(O_(Hm),O_(h)(k−1))O_(w)(k)=max(L₄−L₃,O_(w)(k−1))

As for the horizontal segment representing the side of the vehicle, thepoints of the vertical segment representing the rear and/or the top ofthe vehicle are clockwise angle sorted, so L₄ is the point with thesmallest angle and L₃ is the segment-point with the largest angle.O_(l)(k), O_(h)(k) and O_(w)(k) are respectively the current length,height and width value at time k.

FIG. 27D and FIG. 28D represent detections when the back of the vehicleis fully in the detection FOV. Six side detections and four reardetections are apparent on FIG. 27D while one detection is apparent onFIG. 28D. The width O_(lm) dimension is calculated as follows:O_(lm)(k)=α*(L₄−L₃)+(1−α)*O_(lm)(k−1)

where O_(lm)(k) is the current width at time k and α is the filteringrate.

The size of the vehicle can then be determined fully.

The segmentation algorithm 800 based on a 3D bounding box for selectionof the relevant measures is illustrated in FIG. 29. The first threesteps are identical to that of FIG. 17. If step 120 finds horizontallines, then step 120 is followed by step 121. As explained above, thepoints of a segment are clockwise angle sorted with L₂, the smallestangle and L₁ the largest angle. This segment length is given by L₂−L₁.Otherwise, the next step 123 initializes the 3D bounding box with adefault vehicle length. Step 121 is followed by step 122 which considersthat two segments have a common corner if there is a point ofintersection P_(i) between the two segments with |P_(i)−L₁| and|P_(i)−L₄| less than a distance threshold. If no corner is found, step123 initializes the 3D bounding box with default values. Otherwise, step124 computes the 3D bounding box dimensions from equations presentedabove with respect to FIG. 27C.

It is of interest to derive minimum variance bounds on estimation errorsto have an idea of the maximum knowledge on the speed measurement thatcan be expected and to assess the quality of the results of the proposedalgorithms compared with the bounds. In time-invariant statisticalmodels, a commonly used lower bound is the Cramer-Rao Lower Bound(CRLB), given by the inverse of the Fisher information matrix. The PCRBcan be used for estimating kinematic characteristics of the target.

A simulation was done according to the scenario shown in FIG. 30. Thevehicle 130 is moving at a speed of 60 m/s along a straight line in lane3. The PCRB was applied. As shown in FIG. 31, the tracking algorithmconverges at point 903 at about σ_({dot over (K)}F)=0.48 km/h after 80samples. From point 900, it is apparent that after 16 samples,σ_({dot over (K)}F)<3 km/h, from point 901 that after 28 samples,σ_({dot over (K)}F)<1.5 km/h and from point 902 that after 39 samples,σ_({dot over (K)}F)<1 km/h. Experimental tests confirmed the utility andviability of this approach.

Image Processing and Applications

The multipurpose traffic detection system uses a high-resolution imagesensor or more than one image sensor with lower resolution. In thelatter case, the control and processing unit has to process an imagestitching by combining multiple images with different FOVs with someoverlapping sections in order to produce a high-resolution image.Normally during the calibration process, the system can determine exactoverlaps between images sensors and produce seamless results bycontrolling and synchronizing the integration time of each image sensorand the illumination timing and analyzing overlap sections. Infrared andcolor image sensors can be used with optical filters.

At night, a visible light is required to enhance the color of the image.A NIR flash is not visible to the human eye and does not blind drivers,so it can be used at any time of the day and night.

Image sensors can use electronic shutters (global or rolling) ormechanical shutters. In the case of rolling shutters, compensation forthe distortions of fast-moving objects (skew effect) can be processedbased on the information of the position and the speed of the vehicle.Other controls of the image sensor like Gamma and gain control can beused to improve the quality of the image in different contexts ofillumination.

FIG. 32A is a photograph showing an example snapshot taken by a 5Mpixels image sensor during the day. Vehicles are at a distance ofapproximately 25 m and the FOV at that distance covers approximately 9 m(almost equivalent to 3 lanes). FIGS. 32B, 32C and 32D show the qualityof the image and resolution of FIG. 32A by zooming in on the threelicense plates.

FIG. 33A is a photograph showing an example snapshot taken by the imagesensor at night without any light. This image is completely dark. FIG.33B shows the same scene with infrared light. Two vehicles can be seenbut the license plates are not readable even when zooming in as seen inFIG. 33C. The license plate acts as a retro-reflector and saturates theimage sensing. FIGS. 34A and 34B use the same lighting with a lowerintegration time. The vehicle is less clear but the image shows somepart of the license plate becoming less saturated. FIGS. 34C and 34Ddecrease a little more the integration time and produce a readablelicense plate.

One way to get a visible license plate at night and an image of thevehicle is to process several snapshots with different integration times(Ti). For example, when the 3D detection confirms the position of avehicle in the detection zone, a sequence of acquisition of severalsnapshots (ex.: 4 snapshots with Ti1=50 μs, Ti2=100 μs, Ti3=250 μs andTi4=500 μs), each snapshot taken at a certain frame rate (ex.: each 50ms), will permit to get the information on a specific vehicle:information from the 3D sensor, a readable license plate of the trackedvehicle and an image from the context including the photo of thevehicle. If the system captures 4 images during 150 ms, a vehicle at 150km/h would travel during 6.25 m (one snapshot every 1.5 m).

To enhance the quality of the image, high dynamic range (HDR) imagingtechniques can be used to improve the dynamic range between the lightestand darkest areas of an image. HDR notably compensates for loss ofinformation by a saturated section by taking multiple pictures atdifferent integration times and using stitching process to make a betterquality image.

The system can use Automatic License Plate Recognition (ALPR), based onOptical Character Recognition (OCR) technology, to identify vehiclelicense plates. This information of the vehicle identification andmeasurements is digitally transmitted to the external controller or bythe network to back-office servers, which process the information andcan traffic violation alerts.

The multipurpose traffic detection system can be used day or night, ingood or bad weather condition, and also offers the possibility ofproviding weather information like the presence of fog or snowingconditions. Fog and snow have an impact on the reflection of theradiated light pulses of the protective window. In the presence of fog,the peak amplitude of the first pulse exhibits sizable timefluctuations, by a factor that may reach 2 to 3 when compared to itsmean peak amplitude level. Likewise, the width of the first pulse alsoshows time fluctuations during these adverse weather conditions, butwith a reduced factor, for example, by about 10 to 50%. During snowfalls, the peak amplitude of the first pulse visible in the waveformsgenerally shows faster time fluctuations while the fluctuations of thepulse width are less intense. Finally, it can be noted that along-lasting change in the peak amplitude of the first pulse can besimply due to the presence of dirt or snow deposited on the exteriorsurface of the protective window.

FIG. 35 shows an example image taken with infrared illumination with theoverlay (dashed lines) representing the perimeter of the 16 contiguousdetection zones of the 3DOR. Apparent on FIG. 35 are high intensityspots 140 coming from a section of the vehicle having a highretro-reflectivity characteristic. Such sections having a highretro-reflectivity characteristic include the license plate,retro-reflectors installed one the car and lighting modules that caninclude retro-reflectors. An object with retro-reflectivitycharacteristic reflects light back to its source with minimumscattering. The return signal can be as much as 100 times stronger thana signal coming from a surface with Lambertian reflectance. Thisretro-reflectivity characteristic has the same kind of impact on the3DOR. Each 3D channel detecting a retro-reflector at a certain distancein its FOV will acquire a waveform with high peak amplitude at thedistance of the retro-reflector. The numbers at the bottom of theoverlay (in dashed lines) represent the distance measured by themultipurpose traffic detection system in each channel which contains ahigh peak in its waveform. Then, with a good image registration betweenthe 2D image sensor and the 3D sensor, the 2D information (spot withhigh intensity) can be correlated with the 3D information (highamplitude at a certain distance). This link between 2D images and 3Ddetection ensures a match between the identification data based onreading license plates and measurements of position and velocity fromthe 3D sensor.

The license plate identification process can also be used as a secondalternative to determine the speed of the vehicle with lower accuracybut useful as a validation or confirmation. By analyzing the size of thelicense plate and/or character on successive images, the progression ofthe vehicle in the detection zone can be estimated and used to confirmthe measured displacement.

The embodiments described above are intended to be exemplary only. Thescope of the invention is therefore intended to be limited solely by theappended claims.

We claim:
 1. A method for tracking and characterizing a plurality ofvehicles simultaneously in a traffic control environment, the methodcomprising: providing a 3D optical emitter at an installation heightoriented to allow illumination of a 3D detection zone in saidenvironment; providing a 3D optical receiver oriented to have a wide anddeep field of view within said 3D detection zone, said 3D opticalreceiver having a plurality of detection channels in said field of view;driving the 3D optical emitter into emitting short light pulses towardthe detection zone, said light pulses having an emitted light waveform;receiving a reflection/backscatter of the emitted light on the vehiclesin the 3D detection zone at said 3D optical receiver, thereby acquiringan individual digital full-waveform LIDAR trace for each detectionchannel of said 3D optical receiver; using said individual digitalfull-waveform LIDAR trace and said emitted light waveform, detecting apresence of a plurality of vehicles in said 3D detection zone, aposition of at least part of each said vehicle in said 3D detection zoneand a time at which said position is detected; assigning a uniqueidentifier to each vehicle of said plurality of vehicles detected;repeating said steps of driving, receiving, acquiring and detecting, ata predetermined frequency; at each instance of said repeating step,tracking and recording an updated position of each vehicle of saidplurality of vehicles detected and an updated time at which said updatedposition is detected, with said unique identifier; wherein saiddetecting said presence includes: extracting observations in theindividual digital full-waveform LIDAR trace; using the location for theobservations to remove observations coming from a surroundingenvironment; extracting lines using an estimate line and a covariancematrix using polar coordinates; removing observations located on linesparallel to the x axis.
 2. The method as claimed in claim 1, whereinsaid traffic control environment is at least one of a traffic managementenvironment and a traffic enforcement environment.
 3. The method asclaimed in claim 1, wherein said detecting said presence includesextracting observations in the individual digital full-waveform LIDARtrace and intensity data for the observations; finding at least one blobin the observations; computing an observation weight depending on theintensity of the observations in the blob; computing a blob gravitycenter based on the weight and a position of the observations in theblob.
 4. The method as claimed in claim 1, further comprising setting atleast one trigger line location and recording trigger line trespassingdata with the unique identifier.
 5. The method as claimed in claim 4,further comprising setting said trigger line location relative to avisible landmark in said environment.
 6. The method as claimed in claim1, wherein said detecting said time at which said position is detectedincludes assigning a timestamp for said detecting said presence andwherein said timestamp is adapted to be synchronized with an externalcontroller.
 7. The method as claimed in claim 1, further comprisingobtaining a classification for each detected vehicles using a pluralityof detections in the 3D detection zone caused by the same vehicle. 8.The method as claimed in claim 1, wherein said detecting said presencefurther comprises detecting a presence of a pedestrian in saidenvironment.
 9. The method as claimed in claim 1, wherein said part ofsaid vehicle is one of a front, a side and a rear of the vehicle. 10.The method as claimed in claim 1, wherein emitting short light pulsesincludes emitting short light pulses of a duration of less than 50 ns.11. The method as claimed in claim 1, wherein said 3D optical emitter isat least one of an infrared LED source, a visible-light LED source and alaser.
 12. The method as claimed in claim 1, wherein said providing said3D optical receiver to have a wide and deep field of view includesproviding said 3D optical receiver to have a horizontal field of viewangle of at least 20° and a vertical field of view angle of at least 4°.13. The method as claimed in claim 1, further comprising determining andrecording a speed for each said vehicle using said position and saidupdated position of one of said instances of said repeating step and anelapsed time between said time of said position and said updated time ofsaid updated position, with said unique identifier.
 14. The method asclaimed in claim 13, further comprising using a Kalman filter todetermine an accuracy for said speed to validate said speed; comparingsaid accuracy to a predetermined accuracy threshold; if said accuracy islower than said predetermined accuracy threshold, rejecting said speed.15. The method as claimed in claim 14, further comprising retrieving aspeed limit and identifying a speed limit infraction by comparing saidspeed recorded for each said vehicle to said speed limit.
 16. The methodas claimed in claim 1, further comprising: providing a 2D opticalreceiver, wherein said 2D optical receiver being an image sensor adaptedto provide images of said 2D detection zone; driving the 2D opticalreceiver to capture a 2D image; using image registration to correlatecorresponding locations between said 2D image and said detectionchannels; extracting vehicle identification data from said 2D image at alocation corresponding to said location for said detected vehicle;assigning said vehicle identification data to said unique identifier.17. The method as claimed in claim 16, wherein the vehicleidentification data is at least one of a picture of the vehicle and alicense plate alphanumerical code present on the vehicle.
 18. The methodas claimed in claim 17, wherein the vehicle identification data includessaid 2D image showing a traffic violation.
 19. The method as claimed inclaim 17, further comprising extracting at least one of a size ofcharacters on the license plate and a size of the license plate andcomparing one of said size among different instances of the repeating todetermine an approximate speed value.
 20. The method as claimed in claim16, further comprising providing a 2D illumination source oriented toallow illumination of a 2D detection zone in said 3D detection zone anddriving the 2D illumination source to emit pulses to illuminate said 2Ddetection zone and synchronizing said driving the 2D optical receiver tocapture images with said driving the 2D illumination source to emitpulses to allow capture of said images during said illumination.
 21. Themethod as claimed in claim 20, wherein driving the 2D illuminationsource includes driving the 2D illumination source to emit pulses of aduration between 10 μs and 10 ms.
 22. The method as claimed in claim 19,wherein the 2D illumination source is at least one of a visible lightLED source, an infrared LED light source and laser.
 23. The method asclaimed in claim 19, wherein the 3D optical emitter and the 2Dillumination source are provided by a common infrared LED light source.24. The method as claimed in claim 19, wherein the vehicleidentification data is at least two areas of high retroreflectivityapparent on the images, said detecting a presence includes extractingobservations in the individual digital signals and intensity data forthe observations, the method further comprising correlating locationsfor the areas of high retroreflectivity and high intensity datalocations in the observations, wherein each said area of highretroreflectivity is created from one of a retroreflective licenseplate, a retro-reflector affixed on a vehicle and a retro-reflectivelighting module provided on a vehicle.
 25. The method as claimed inclaim 16, further comprising combining multiples ones of said capturedimages into a combined image with the vehicle and the vehicleidentification data apparent.
 26. A vehicle detection system fortracking and characterizing a plurality of vehicles simultaneously in atraffic control environment, the system comprising: a 3D optical emitterprovided at an installation height and oriented to allow illumination ofa 3D detection zonein the environment; a 3D optical receiver providedand oriented to have a wide and deep field of view within the 3Ddetection zone, the 3D optical receiver having a plurality of detectionchannelsin said field of view; a controller for driving the 3D opticalemitter into emitting short light pulses toward the detection zone, thelight pulses having an emitted light waveform; the 3D optical receiverfor receiving a reflection/backscatter of the emitted light on thevehicles in the 3D detection zone, thereby for acquiring an individualdigital full-waveform light detection and ranging (LIDAR) trace for eachchannel of the 3D optical receiver; a processor configured for detectinga presence of a plurality of vehicles in the 3D detection zone using theindividual digital full-waveform LIDAR traceand the emitted lightwaveform, capturing a series of vehicle measurements from the LIDARtrace, detecting a position of at least part of each the vehicle in the3D detection zone, recording a time at which the position is detected,and assigning a unique identifier to each vehicle of the plurality ofvehicles detectedand tracking and recording an updated position of eachvehicle of the plurality of vehicles detected and an updated time atwhich the updated position is detected, with the unique identifier; a 2Doptical receiver, wherein the 2D optical receiver is an image sensoradapted to provide images of the 2D detection zone; and a driver fordriving the 2D optical receiver image sensor to capture a 2D image theimages; the processor being further adapted configured for using imageregistration to correlate corresponding locations between said 2D imageimages and said detection channels, estimating a length of a vehicle byfitting a first line to a first subset of said series of vehiclemeasurements, estimating a width of a vehicle by fitting a second lineto a second subset of the vehicle measurements, and extracting vehicleidentification data from the 2D image at a location corresponding to thelocation for the a detected vehicle; and assigning the vehicleidentification data to the unique identifier.
 27. The system as claimedin claim 26, wherein said processor is further for determining andrecording a speed for each the vehicle using the position and the anupdated position of one of the instances of the repeating step and anelapsed time between the time of the position and the an updated time ofthe updated position, with the unique identifier.
 28. The system asclaimed in claim 26, further comprising a 2D illumination sourceprovided and oriented to allow illumination of a 2D detection zone inthe 3D detection zone; a source driver for driving the 2D illuminationsource to emit pulses; a synchronization module for synchronizing saidsource driver and said driver to allow capture of said images while said2D detection zone is illuminated.
 29. A method for tracking andcharacterizing a plurality of vehicles simultaneously in a trafficcontrol environment, the method comprising: providing a 3D opticalemitter at an installation height oriented to allow illumination of a 3Ddetection zone in said environment; providing a 3D optical receiveroriented to have a wide and deep field of view within said 3D detectionzone, said 3D optical receiver having a plurality of detection channelsin said field of view; driving the 3D optical emitter into emittingshort light pulses toward the detection zone, said light pulses havingan emitted light waveform; receiving a reflection/backscatter of theemitted light on the vehicles in the 3D detection zone at said 3Doptical receiver, thereby acquiring an individual digital full-waveformLIDAR trace for each detection channel of said 3D optical receiver;using said individual digital full-waveform LIDAR trace and said emittedlight waveform, detecting a presence of a plurality of vehicles in said3D detection zone, a position of at least part of each said vehicle insaid 3D detection zone and a time at which said position is detected;assigning a unique identifier to each vehicle of said plurality ofvehicles detected; repeating said steps of driving, receiving, acquiringand detecting, at a predetermined frequency; at each instance of saidrepeating step, tracking and recording an updated position of eachvehicle of said plurality of vehicles detected and an updated time atwhich said updated position is detected, with said unique identifier,wherein said detecting said presence includes: extracting observationsin the individual digital full-waveform LIDAR trace and intensity datafor the observations; finding at least one blob in the observations;computing an observation weight depending on the intensity of theobservations in the blob; computing a blob gravity center based on theweight and a position of the observations in the blob.
 30. The vehicledetection system of claim 26, wherein the processor is furtherconfigured to estimate a height of the vehicle.
 31. The vehicledetection system of claim 26, wherein the processor is configured toestimate a volume of the vehicle based at least in part on the vehiclemeasurements.
 32. The vehicle detection system of claim 26 wherein theprocessor is configured to identify a corner point of the vehicle lessthan a threshold distance from points on both of the first and secondlines.
 33. The vehicle detection system of claim 32, wherein theprocessor is configured to define a three-dimensional bounding boxcorresponding to the vehicle based on detection of corners.
 34. Thevehicle detection system of claim 33, wherein the three-dimensionalbounding box represents an estimate of bounding dimensions of thevehicle.
 35. The vehicle detection system of claim 34, wherein theprocessor is further configured to refine the estimate of the boundingdimensions as the light detection and ranging (LIDAR) trace is producedby reflection of the illumination signals from an increasing number ofsides of the vehicle.
 36. The vehicle detection system of claim 26,wherein the light detection and ranging (LIDAR) trace includesreflection of the illumination signals from a complete side of thevehicle, and wherein the processor is configured to determine dimensionsof a three-dimensional bounding box corresponding to the vehicle basedat least on full-waveform signal processing of the signal waveforms fromthe vehicle measurements for a complete side of the vehicle.
 37. Thevehicle detection system of claim 26, wherein the processor isconfigured to account for changes in the distance between the vehicleand the 3D optical emitter due to relative movement between the 3Doptical emitter and the vehicle.
 38. The vehicle detection system ofclaim 26, wherein the processor is configured to assign a classificationto the vehicle based on a dimension of the vehicle.
 39. The vehicledetection system of claim 38, wherein the classification is according toa classification scheme distinguishing two-wheeled and four-wheeledvehicles.
 40. The vehicle detection system of claim 26, wherein theprocessor is configured to trigger an event based at least in part on adimension of the vehicle.